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© This invention relates to an image display ap- 
paratus having first image generating means for dis- 
playing first image information by a plurality of pixels 
emitting light, second image generating means for 
displaying second image information by a plurality of 
pixels emitting light, the second image information 
being part of the first image information, visual axis 
detecting means for detecting an observer's visual 
axis direction, optical means for combining a first 
image from the first image generating means and a 
second image from the second image generating 



means, moving the second image relative to the first 
image m conformity with an output signal from the 
visual axis detecting means and turning it to the 
observer's pupil, the marginal portion of the second 
image being made substantially coincident with the 
boundary between the pixels of the first image when 
the images are combined, and image display chang- 
ing means for changing the first image information 
and the second image information on the basis of 
the output signal of the visual axis detecting means. 



Rank Xerox (UK) Business Services 

•3 X/3.09/3 3 «' 




2 



J 



£uropeui Pairnt 
Office 



EUROPEAN SEARCH REPORT 



AjfUCUtoo Ntab« 

EP 94 10 4880 



Category 



51 

IL 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Ckaboa of document with 
.. of rde>aot pnagi 



where appropriate. 



Rear* wl 
to daim 



GB-A-2 259 213 (BRITISH AEROSPACE) 3 March 
" claims 1,5,8 * 

US-A-4 479 784 (KALLINSON RICHARD B ET 
AL) 30 October 1984 

" claim 6 * 

US-A-4 348 186 (HARVEY JAMES F ET AL) 7 
September 1982 

• claims 9-11; figure 2 * 

US-A-4 439 157 (BREGLIA DENIS R ET AL) 27 
March 1984 

" abstract; figures 1,3 * 

US-A-5 106 179 (KAMAYA NAOKI ET AL) 21 
April 1992 

* claims 1,2; figures 3,4 * 



1.9-12, 
25,36 



1,3,5,7, 
8,12,25, 
36 



1.12,25, 
36 



7,8 



21,45 



CLASSIFICATION OF THE 
APPLICATION OblCLS) 



G02B27/00 
G09B9/30 



The prcscat seven report has bcea dYawo up for all dai 



BERLIN 



1 September 1995 



TECHNICAL FIELDS 
SEARCHED OM.a.5) 



G02B 
G09B 



Fuchs, R 



CATEGORY OF CITED DOCUMENTS 

X : avnailuly r^mei if taka Uom 

Y : aarbailaHy raiovut if coabta«4 vith uotacr 

aocaaiAi of ta« itai caiogory 
A : taduoiogicaJ aadiroua^ 
O : nM.«rm«B Jisdosun 
P : iataraoiiaic docuacst 



T : theory or pnaaptt uo4«r)yut| the tevtatioe 
E : mriw ptiwt tfoaioMot, tut puaiisbH oe, or 

afiar tt* Aliag tut 
D : tfoauaaat di*4 i» tat tppJicatJoe 
L : «oaitt«Bi diH for otW rmioas 



* : naWr of tb« 



er of tb« nmt aaicoi taiairy, corrapoetftag 



£P CMcWl (3) A 

g o r ; . : ? / o o ri ^ - ( & o b c ) 



0 EUROPEAN PATENT APPLICATION 

0 Application number: 94104880.3 © Int. CI. 5 G02B 27/00 



© Date of filing: 28.03.94 



© Priority: 29.03.93 JP 69937/93 


Ohta-ku, Tokyo 146 (JP) 


16.07.93 JP 176777/93 


Inventor: Kobayashi, Shin 


1 fi 03 94 JP 48645/94 


c/o Canon K.K., 

ou-i, j-cnome bnimomaruKo 


© Date of publication of application: 


unia-Ku, lOKyo i4b (Jr) 


05.10.94 Bulletin 94/40 


Inventor: Sudo, Toshiyuki 
c/o Canon K.K., 


© Designated Contracting States: 


30-2. 3-chome Shimomaruko 


DE ES FR GB IT NL 


Ohta-ku, Tokyo 146 (JP) 


© Applicant: CANON KABUSHIKI KAISHA 


Inventor: Morishima, Hideki 


c/o Canon K.K., 


30-2, 3-chome, Shimomaruko, 


Ohta-ku 
Tokyo (JP) 


30-2, 3-chome Shimomaruko 


Ohta-ku, Tokyo 146 (JP) 


Inventor: Kaneko. Tadashi 


© Inventor: Taniguchi, Naosato 


c/o Canon K.K., 


c/o Canon K.K., 


30-2, 3-chome Shimomaruko 


30-2, 3-chome Shimomaruko 


Ohta-ku, Tokyo 146 (JP) 


Ohta-ku, Tokyo 146 (JP) 




Inventor: Matsumura, Susumu 


© Representative: Pellmann, Hans-Bernd, 


c/o Canon K.K., 


30-2, 3-chome Shimomaruko 


Dipl.-lng. 


Ohta-ku. Tokyo 146 (JP) 


Patentanwaltsburo 


Inventor: Yoshinaga, Yoko 


Tiedtke-Buhling-Kinne & Partner 


c/o Canon K.K., 


Bavariaring 4 


30-2, 3-chome Shimomaruko 


D-80336 Munchen (DE) 



j Publication number 



lllllllllllllllll 

0 618 471 A2 



© Image display apparatus and image photographi 

© This invention relates to an image display ap- 
paratus having first image generating means lor dis- 
CJ playing first image information by a plurality of pixels 
^ emitting light, second image generating means for 
^ displaying second image information by a plurality of 
ps* pixels emitting light, the second image information 
^ being par: of the first image information, visual axis 
qq detecting means for detecting an obsetver's visual 
axis direction, optical means for combining a first 
t& image from the first image generating means and a 
O second image from the second image generating 

CL 
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ng apparatus therefor. 

means, moving the second image lelative to the first 
image in conformity with an output signal from the 
visual, axis detecting means and turning it to the 
observer's pupil, the marginal portion of the second 
image being made substantially coincident with the 
boundary between the pixels of the first image when 
the images are combined, and image display chang- 
ing means for changing the lust image information 
and the second image information on the basis of 
the output signal of the visual axis detecting means. 
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BACKGROUND OF TH E INVENTION 

Field of the Invention 

This invention relates to an apparatus lor dis- 
playing an image of a wide field angle, and particu- 
larly to an image display apparatus adapted to be 
mounted on the head portion or face of a person 
who observes an image and enable the person to 
observe the image information of a small display 
element such as a liquid, crystal display element or 
a CRT therethrough while enlarging the image in- 
formation as a virtual image. 

Related Background Art 

A display apparatus using a multilayer film 
reflecting surface or an optically transparent beam 
coupling element such as a hologram optica! ele- 
ment to enable display information (image informa- 
tion) from a display element and image information 
such as a scene in the external world to be spa- 
tially superposed one upon the other in the same 
field of view and observed is generally called a 
head up display apparatus and is utilized in various 
fields. Also, a display apparatus in which the beam 
combiner is provided at a location relatively near to 
an observer's eyes in order to make the parts of 
the optical device of the head up display apparatus 
small and the optical device is mounted on a 
helmet is called a helmet mounted display appara- 
tus (HMD apparatus), and its various applications to 
a display apparatus for the operation of an aircraft, 
a display apparatus lor amusements such as 
games or for virtual reality, etc. have been pro- 
posed. 

Fig. l of the accompanying drawings is a sche- 
matic view of an HMD apparatus as a display 
apparatus described in U.S. patent No. 4.028.725. 
Display lights emitted from CRTs 101 and 102 aie 
made into substantially parallel beams of light by 
collimator lenses 103 and 1 08 and are reflected by 
mirrors 104 and 107. respectively, and the display 
lights Irom the two CRTs 1 0 1 and 102 are super- 
posed one upon the other by dichroic mirrors 105 
and 109. The display lights thus superposed one 
upon the other have their direction agam changed 
by a mirror 106 and are enlarged through a lens 
1 10. and are directed to an observer's pupil 1 1 4 by 
a half mirror 111. At this time, the CRT 1 01 dis- 
plays a wide field image 113 and the CRT 1 02 
displays a nanow field image 112. Also, the dich- 
roic mirror 109 is installed so as to shield the 
vicinity ol the center of the dichroic mirroi 105 fiom 
iight and therefore, the observer can observe the 
wide field image 113 displayed by the CRT 101 
and the narrow field image 1 12 from the CRT 102 
displayed near the center thereof while superpos- 



ing these images upon a scene o> the- line benmc 
the half mirror 1 1 1 

Also/ m this example of the pnor an. a so- 
called Durkmje image provided by the iignt from a 
5 light source 115 of infrared rays being reflected by 
the observer's puoii 114 is detected by a detection 
unit 119 comprising a beam splitter 116 and 
photodioaes 117. 118. whereby the observer's vi- 
sual axis direction is detected 

w The output of this visual axis detecting means 

is inputted to the servo system (not shown) of the 
movable half mirror 1 1 1 and thus, the aforemen- 
tioned two field images are always displayed for* 
wardly in the observer's visual axis direction. 

'5 In the example of the conventional art shown in 

Fig. i, the field angles of the wide field image and 
the narrow field image are 25* and 5 *, respec- 
tively, and the narrow field image is displayed with 
a high resolution, whereby with the displayed im- 

20 age of the high resolution as the center of the field 
of view, the wide field image is always observed 
around it. In the above-described example of the 
conventional art. however, design is made such 
that the image formed by the wide field image and 

25 the narrow field image being superposed one upon 
the other is always presented at the central portion 
of the field of view by the half mirror 1 1 1 controlled 
by the signal from the visual axis detecting means, 
and this has led to a problem that even if the 

30 observer gazes at any point on the wide field 
image, the image is not visually recognized at the 
center of the field of vrew and the portion intended 
by the observer cannot be observed Also, a field 
angle much wider than the field angle as in the 

35 above-described example of the prior art, e.g. 25'. 
is required of the display apparatus for virtual re- 
ality and for example, a display for displaying a 
field angle of 88' by one eye has been commer- 
cialized, but in these apparatuses, the resolution of 

•to the displayed image depends on the pixel density 
of the liquid crystal display element used, and this 
has led to a disadvantage that at present, the 
displayed image does not have a sufficient iesolu- 
tion. 

SUMMARY OF THE INVENTIO N 

It is the object of the present invention to 
provide an image display apparatus for enabling an 

so observer to observe the image of a place the 
observer wants to seen in a wide field image by a 
highly line image, and an image photographing 
apparatus therefor. 

To achieve the above object, one form of the 

55 image display apparatus of the present invention is 
characterized by first image generating means fo» 
displaying first image information by a plurality of 
pixels emitting light, second image generating 



3 



EP 0 618 471 A2 



means for displaying second image information by 
a plurality of pixels em.mng light, the second .mage 
information being pan of the first image informa- 
tion, visual ax.s detecting means for detecting an 
observer's visual axis direction optical means for 5 
combining a first .mage from the first image gen- 
erating means and a second .mage from the sec* 
ond image generating means, moving the second 
•mage relative to the first image in conformity with 
an output signal from the visual axis detecting w 
means and turning it to the observer's pupil, the 
marginal -portion of the second image being made 
substantially coincident with the boundary between 
the pixels of said first image when the images are 
combined and image display changing means for ; 5 
changing said first image information and the sec- 
ond image information on the basis of the output 
signal of the visual axis detecting means. 

In a further preferred form, the size of the 
second .mage is integer times as great as the 20 
interval between the pixels (or the size of the pixel) 
of the first .mage when the images are combined. 

In still a further preferred form, the first image 
information in an area wherein the first image and 
the second image are superposed one upon the 25 
other when the images are combined is the second 
image information. 

fn yet still a further preferred form, the area of 
the images superposed one upon the other is not 
displayed by the first image generating means. 30 

In a further preferred form, the pixel density in 
the central portion of the second image is higher 
than the pixel density of the first .mage and the 
pixel density in the marginal portion of the second 
-mage is substantially the same as the pixel density 35 
of the first .mage when said images are combined. 

In still a further preferred form, the luminance 
m the central portion of the second image is higher 
than the luminance of the first image and the 
luminance in the marginal portion of the second * 0 
'mage is substantially the same as the luminance 
of the first image when the .mages are combined. 

In yet sttll a further form, the first and second 
images are combined on an intermediate imaging 
plane thtough respective relay lenses, are enlarged ,- 5 
as virtual images and are turned to the observers 
pupil. 

In a further preferred form, the relay lenses 
have different imaging magnifications. 

Another form of the .mage d.splay apparatus of 50 
the present invention is characterized m that the 
'mage display apparatus is provided correspond- 
ingly to each of the observer's left and right eyes. 

In a furthei preferred form, the second .mage 
generating means corresponding to the left and 55 
nght eyes display parallax .mage mformat.on cor- 
responding to the respective eyes. 



fn still a further preloned form, the first image 
generaimg means is used m common 

Anotne- form of the image display apparatus of 
the present invention is characterized by first im- 
age generating means for emitting light and dis- 
playing fust image information second image gen- 
erating means for emitting hght and displaying sec- 
ond image information, the second image informa- 
tion being part of sa.d first image information visual 
axis detecting means for detecting an observer's 
visual axis direction head portion position detecting 
means for detecting the position of the observer's 
head portion optical means for combining a first 
image from the first image generating means and a 
second .mage from the second image generating 
means, mov.ng the second .mage relative to the 
first image in conformity with an output signal from 
the visual axis detecting means and turning it to 
the observer's pupil and image information chang- 
ing means for changing the first image information 
and the second image information on the basis of 
the output signal of the visual axis detecting means 
and the output signal of the head port.on position 
detecting means. 

A further preferred form has an image memory 
having recorded therein the image all around a 
predetermined position, and inputs a part of said 
•mage memory to sa.d first image generating 
means as the first image information m conformity 
with the output signal of the head portion position 
detecting means. 

One form of the image photographing appara- 
tus of the present invention is an image photog- 
raphing apparatus for an .mage display apparatus 
for displaying, on the basis of output s.gnals from 
visual ax.s detecting means for detect.ng the visual 
axis direction of an observer who observes a first 
image from first image generating means for dis- 
playing first image information and head portion 
pos.t.on detect.ng means for detecting the position 
of the observer's head portion, a combined image 
of said first .mage and a second image Irom sec- 
ond image generating means for display.ng second 
•mage mformat.on which .s part of the first -mage 
-nformat.on. character.zed by first .mage pickup 
means for obtaining the first .mage information 
second .mage p.ckup means for obtaining the sec- 
ond image information means for controlling the 
photo-taking direction of the first .mage p.ckup 
means m conformity with the output s.gnal from the 
head port.on pos.t.on detecting means and means 
for controlling the photo-tak.ng direction of the sec- 
ond image pickup means m conformity with the 
output signal from the visual axis detecting means. 

A further preferred form has means for trans- 
mitting the image information obtained from the 
first image pickup means to the image display 
apparatus while making the compression rate of 
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the image information higher than that of the image 
information obtained from said second image pic- 
kup means. 

In still a further preferred form, the image 
photographing apparatus is provided correspond- 
ingly to each of the observer's left and right eyes. 

In yet still a further preferred form, said first 
image pickup means is used in common. 

Some specific embodiments of the image dis- 
play apparatus of the present invention will become 
apparent from the following detailed description 
thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows an example of the image display 
apparatus according to the conventional art. 

Fig. 2 shows a first embodiment of the image 
display apparatus of the present invention. 

Figs. 3A. 3B. 3C and 3D are views for illustrat- 
ing images visually recognized by an observer. 

Fig. 4 is a view showing an image visually 
recognized by the observer. 

Figs. 5A. 5B, 5C and 5D are views for illustrat- 
ing image processing. 

Figs. 6A and 6B are views for illustrating image 
processing. 

Fig. 7 is a view for illustrating the pixel arrange- 
ment of a combined image. 

Fig. 8 is a view for illustrating the pixel arrange- 
ment of the combined image. 

Figs. 9A and 9B are views for illustrating the 
pixel arrangement of the combined image. 

Fig. 10 is a view for illustrating a smoothing 
method. 

Fig. 11 is a view for illustrating the pixel ar- 
rangement of the combined image. 

Fig. 12 shows a second embodiment of the 
image display apparatus of the present invention. 

Figs. 13A. 13B ana 13C are views for illustrat- 
ing the pixel arrangement of a combined image. 

Fig. 14 shows a third embodiment of the image 
display apparatus of the present invention. 

Fig. 15 shows a fourth embodiment of the 
image display apparatus of the present invention. 

Fig. 16 shows a fifth embodiment of the image 
display apparatus of the present invention. 

Fig. 17 is a view lor illustrating the relations 
between a head portion position detection signal 
and a visual axis detection signal. 

Figs. 18A. 18B and 18C are views for illustrat- 
ing the relations between a detection signal and an 
image display state. 

Fig. 19A shows a first embodiment of the im- 
age photographing apparatus of the present inven- 
tion. 

Fig. 19B shows a modification of the first em- 
bodiment of the image photographing apparatus of 
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the piesent invention. 

Fig 19C shows a second embodiment of the 
image photographing apparatus of the present in- 
vention. 

5 

DETAILED DESCRIPTI O N QF THE PREFERR ED 
EMBODIMENTS " 

Fig. 2 is a schematic diagram showing a first 

io embodiment of the image display apparatus of the 
present invention. Light from a back light source 1 
such as a fluorescent lamp is modulated by an 
image display element 2 such as a liquid crystal 
display element, and for example, a beam of light 3 

rs emitted from a point a is displayed. This beam of 
light 3 passes through a half mirror 16, is con- 
verted into a beam of light 5 by a lens 4 and enters 
an observer's pupil 6. At this time, the power of the 
lens 4 and an optical device are set to suitable 

20 values, whereby the observer observes the beam 
of light 5 as a beam of light 7 emerging from a 
point a*. Likewise, a beam of light 8 from a point b 
on the image display element 2 is converted into a 
beam of light 9 by the lens 4. and the observer 

25 observes the beam of light 9 as a beam of light 10 
emerging from a point b\ 

Further, light from a back light source 12 is 
modulated by an image display element 13 such 
as a liquid crystal display element, and for exam- 

30 pie. a beam of light 14 emitted from a point c is 
displayed. This beam of light 14 is reflected by a 
half mirror 16 having drive means 15 movable on 
two axes, and enters the lens 4 and becomes a 
beam of light 17. which enters the observer's pupil 

35 6. Accordingly, the observer observes the beam of 
light 17 as a beam of light emerging from a point 
c\ Likewise, a point d on the image display ele- 
ment is observed as a point d' on a virtual image 
plane 1 1 . 

•io As described above, the observer can observe 

while superposing the image information of the 
image aisplay element as a virtual image 2' and 
the image information of the image display element 
13 as a virtual image 13* one upon the other on the 

-J5 imaging position 1 1 of the virtual image and enlarg- 
ing the two images. 

These two virtual images 2* and 13* form a 
combined image in which the different image in- 
formation 13* is superposed upon a cirtain portion 

50 on the image information 2' as the background, and 
the image information 13' is displayed at any loca- 
non on the image mfoimation 2" by the half mirror 
16 being rotativeiy controlled on two axes by the 
drive means 15 controlled by a positional signal 

55 from a controller 26. 

The displayed position of the image informa- 
tion 13' will now be described. The image display 
apparatus according to the present invention has a 
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visual axis direction delecting unit 35 

As the visual axis detecting method, use can 
be made of one of conventional methoas such as 
the search coil method, the corneal reflection meth- 
od and the limbus tracking method, but the search 
coil method requires the mounting of a strong film 
contact lens (larger than ordinary contact lenses) 
having a coil wound therein and is unsuitable for 
practical use. and the limbus tracking method is a 
method of measuring by detecting the difference in 
the reflectance of illumination light illuminating the 
vicinity of the boundary between the white (sclera) 
and the ins (cornea) of the eye between the re- 
spective portions, and has the advantage of being 
capable of constructing the system of an infrared 
light source and two photodetector elements, but 
suffers from a problem that measurement in the 
vertical direction is difficult. 

Therefore, the present embodiment adopts a 
detecting method using the corneal reflection meth- 
od which is simplest in construction and which has 
sufficient accuracy. 

In Fig. 2. the reference numeral 18 designates 
a light source such as a light emitting diode which 
emits invisible infrared light to the observer. The 
light source 18 is disposed on the focal plane of a 
light projecting lens 19. 

The infrared light emitted from the light source 
18 is made into a parallel beam of light 20 by the 
light projecting lens 19, passes through a half mir- 
ror, is reflected by the half mirror 16. and enters 
ana illuminates the observer's pupil 6 through the 
lens 4. At this time, a corneal reflection image 
based on part of the infrared light reflected by the 
surface of the cornea (not shown) of the eyeball 
again returns along the aforedesenbed optical path, 
is reflected by the half mirror 22 and becomes a 
beam of light 21. which is condensed by a light 
receiving lens 23 and forms a corneal reflection 
image on an image sensor 24. Also, beams of 
reflected light from the two end portions of the iris 
(not shown) likewise form an image on the image 
sensor 24 by the light receiving lens 23. 

These unique points (the corneal reflection im- 
age and the end portions of the ins) are detected 
and subjected to a suitable calculation process, 
whereby the angle of rotation of the optical axis of 
the eyeball can be founa. For example, by 

p-L-sinti = (Za + Zb)2-Zd. 

the angle of rotation v of the ooticai axis ol the 
eyeball can be found. In the above expression, p is 
the imaging magnification of each unique point 
onto the image sensor 24. L .s the distance from 
the center of curvature of the cornea to tne ins. and 
Za. Zb and Zd are the positions of the two end 
portions of the tr.s and the cornea reflection image. 



respectively, on the image sensor 24. 

A visual axis detection signal from the visual 
axis direction detecting unit 35 constructed on the 
basis ol the principle as described above is sut> 
5 jected to a suitable calculation process by a detec- 
tion circuit 25, and a visual axis direction output 
signal is inputted to the controller 26. 

This visual axis direction signal is inputted as 
the positional signal of the virtual image 13' to a 
ic drive circuit 27. and the inclination of the half mirror 
16 is controlled by the drive means 15 rotatabie m 
the directions of two axes (horizontal and vertical 
directions), and the virtual image 13' is displayed 
always in coincidence with the visual axis direction. 
is If the thickness of an optically transparent member 
constituting the half mirror 16 is great, the dis- 
played position of the image 2' will be slightly 
moved by the rotation of this half mirror 16 and 
therefore, it is desirable that the thickness of the 
20 half mirror 16 be as small as possible. 

In the present embodiment, in order to prevent 
the beam of light from the back light source 12 oi 
1 from entering the visual axis direction detecting 
unit 35 as an unnecessary beam of light, an optical 
25 filter absorbing or reflecting visible light and trans- 
mitting only infrared light therethrough can be pro- 
vided at a suitable location between the half mirror 
22 of the visual axis direction detecting unit 35 and 
the half mirror 16 to thereby prevent the incidence 
30 of unnecessary light onto the image sensor 24 and 
enhance the accuracy of the detection signal. 

Also, m the present embodiment, there has 
been shown only a case where only the visual axis 
direction detecting unit is provided to thereby con- 
35 trol the position at which the virtual image 13' is 
displayed, but it is also possible to control the 
displayed position of the virtual image 13* by the 
positional signal of the observer's head portion and 
the visual axis detection signal, by the use of a 
<o suitable head portion position detecting method 
(which will be described later in detail), for exam- 
ple, a method of forming a magnetic field around 
the head portion and providing a magnetic sensor 
on the head portion. 

Particularly. »n the detection of the visual axis 
direction using the corneal reflection method, the 1 
mm movement of the reflected image corresponds 
to 10 degrees of eyeball motion and therefore, if 
the position of the head moves by 1 mm m parallel 
so (without rotating), a detection error of 10 degrees 
will occir. Therefore, it is desired that the signal of 
tne head portion position detecting means be used 
to cancel it. As a matter of course, use can also be 
made of a conventional method utilizing the re- 
55 fleeted image from the rear surface of the cry- 
stalline lens (called the fourth Purk.n,e image) to 
cancel the movement of the head. As will be de- 
scribed later, however, the present invention is 
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characterized in that the image information 2* as 
the background is vaned by the signal (torn the 
head portion position detecting means, and the 
head portion position is always detected from this 
character and therefore, a visual axis detecting 
method using a simple corneal reflection method 
using the first Purkmje image will suffice. 

As described above, the image display appara- 
tus according to the present invention has means 
for relatively moving the image information from 
the two image display elements 2 and 13 on the 
imaged position 1 l of the virtual image, whereby 
different image information 13* can be displayed at 
any position on the image information 2' as the 
background. 

At this time, the image information 13' has its 
position controlled by the detection signal from the 
visual axis direction detecting unit 35 and is dis- 
played always in coincidence with the visual axis 
direction. 

The image information 13' is part of the image 
information 2* and by making its image quality 
highly fine, the observer can highly finely observe 
the image information 13* in the portion of the 
image information 2' of a wide field angle the 
observer wants to see (the vicinity of the gaze 
point). 

Also, the image information 2* as the back- 
ground is varied by the signal from the head por- 
tion position detecting means and further, the gaze 
point on the image information 2' is extracted by 
the signal from the visual axis direction detecting 
unit and the display information of the image dis- 
play element 13 is changed, whereby the observer 
can move his or her head and eyes to thereby 
observe image information which is very wide. e.g. 
ranges over 360 degrees around the observer as if 
all of the image information was displayed highly 
finely. Of course, the order of the rewriting of the 
above-described two images is sometimes con- 
verse as when the gazing point has moved and 
thereafter the head turns m that direction. 

Generally, in equilibrium with yield, an LCD" 
having a small pixel size and a large display area 
is difficult and expensive to manufacture. In the 
present embodiment, the LCD 2 is an LCD having 
a large display area and a large pixel size, and the 
LCD 13 is an LCD having a small dispiay aiea and 
a small pixel size, and when the display area of the 
LCD 2 is S2 and the display area of the LCD 13 is 
S13. the screen size of the LCD 2 and the screen 
size of the LCD 13 are approximately ^S2 s Si3 . 
Also, the LCD 2 and the LCD 13 are optically at the 
same distance relative to the lens 4. and the ratio 
between the sizes of images 2' and 13' eniargedly 
projected onto the virtual image plane 11 are also 
approximately N S2 \ ST3 . Accordingly, the ob- 
server can visually recognize the image of the LCD 



2 as a large image 2' of coaise pixels or. me vinua: 
image piane and the image of the LCD 13 as a 
small image 13' of fine pixels on the virtual image 
plane whue superposing the two images one upon 
5 the other. 

Description will now be made of image sianals 
sent to the LCD 2 and LCD 13 and an image 
visually recognized by the observer 

Figs. 3A to 3D illustrate the images visually 
w recognized by the observer in the present embodi- 
ment. Fig. 3A shows the original image information, 
and Fig. 3B shows image information of the original 
image information of Fig. 3A which is displayed by 
the LCD 13. When the cutting-out of the image is 
75 to be effected by the head portion position and 
direction detecting means, the image information of 
Fig. 3A becomes the image information of the 
widely displaying image area 2'. 

The controller 26 shown m Fig. 2 receives the 
20 visual axis position signal from the detection circuit 
25 and cuts out of the original image information 
Fig. 3A. an image of a size corresponding to the 
display area of the LCD 13 when the visual axis 
position from within the original image information 
25 ts the center and is imaged on the virtual image 
plane 11. and makes it into partial image informa- 
tion Fig, 3A. The controller 26 shown in Fig. 2 
sends the original image information Fig. 3A to a 
drive circuit 28 and sends the image information 
30 Fig. 3B to a drive circuit 29. and the drive circuits 
28 and 29 use the sampling or the like of image 
data by a method similar to that conventionally 
used for the driving of LCD to display images on 
the LCD 2 and LCD 13. respectively. 
35 Fig. 3C shows the image actually displayed on 

the LCD 2 shown in Fig. 2. and Fig. 3D shows the 
image actually displayed on the LCD 13. and the 
LCD 2 of Fig. 2 has large pixels and a large display 
area and therefore. Fig. 3C is a coarse image 
having a great angle of display field, and the LCD 
13 of Fig. 1 has fine pixels and a small display 
area and therefore. Fig. 3D. is a highly fine image 
having a small angle of display field. The dnve 
circuit 27 of Fig. 2 effects the two-axis rotation of 
45 the half mirror 16 m accordance with the aforemen- 
tioned visual axis position signal, and gives the half 
mirror driving system 15 such a dnve signal that 
the center of the virtual image of the LCD 13 by 
the lens 4 coincides with the observers visual axis, 
so Thus, the observer visually recognizes on the vir- 
tual image plane 1 1 the image of Fig. 3C displayed 
on the LCD 2 and the image of Fig. 3D displayed 
on the LCD 13 while superposing these two images 
one upon the other, and the centei of the image 
55 Fig. 3D displayed on the LCD 13 coincides with the 
observer's visual axis. Fig. 4 shows the image 
visually recognized by the observer in the present 
embodiment, and the center of the virtual image of 
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the image displayed on the LCD 13 by the lens 4 
is coincident with the observers visual axis More- 
over, the tmages displayed on the LCD 2 and LCD 
13 are superposed one upon the other by the 
above-desenbed processing and are visually recog- 
nized by the observer, and a highly fine image can 
be observed near the gazing point. 

If at this time, the drive circuit 29 and the 
illumination light source 12 are set so that the 
image to be displayed on the LCD 13 may be 
displayed with high luminance, the visual recogniz- 
ability of the highly fine image near the gazing 
point could be improved. Further, if such a mea- 
sure that the displayed image on the LCD 13 
gradually becomes lower in luminance from the 
center toward the marginal portion thereof is adopt- 
ed, the luminance difference in the boundary por- 
tion between the narrow image 13' and wide image 
2* projected onto the virtual image plane II could be 
made inconspicuous. Specifically, the drive circuit 
29 can be set such that the image signal of the 
image displayed on the LCD 13 becomes lower in 
luminance from the center toward the marginal 
ponion, or an optical filter such as an ND filter 
which becomes higher in light absorption rate from 
the central portion toward the marginal portion can 
be inrtalied in the LCD 13. 

As a basic method of solving the difficulty for 
the image to be seen in the boundary of the 
superposed portion of the images, there is a meth- 
od which will herein after be described, besides the 
above-described method. Figs. 5A to 5D. 6A and 
6B are views for illustrating the processing of the 
images in an embodiment of the present invention 
wherein overlap is prevented. However, in Figs 5C 
and 5D and Fig. 6B. for illustration, pixels are 
oepicted more coarsely than tn the actual embodi- 
ment. 

Fig. 5A shows the original image information 
which is about to be displayed. Fig. 6A shows the 
image information sent to the LCD 2. Fig 5B 
shows the image information sent to the LCD 13. 
and Fig. 5C shows the combined image visually 
recognized by the observer. In Fig 6A. a black- 
colored portion (E> is a portion m which the LCD 2 
displays perfectly black (does not display any im- 
age). The original image information Fig 5A is sent 
to the controller 26 of Fig. 2 by a signal line, not 
shown. The controller 26 receives the visual axis 
position signal from the detection circuit 25. divides 
the original image information Fig. sa into the 
center as the visual axis position m tne original 
image information, the image Fig 56 to be dis- 
played on the LCD 13. and the image information 
Fig. 6A of said image in which of the original image 
information Fig. 5A. the portion corresponding to 
the image information Fig. SB is a black signal 
(luminance signal 0). and sends them to the drive 



circuits 29 and 28 as the image Signals B and A 0 I 
Fig. 2. respectively. 

The onve circuits 29 and 28 effect the sam- 
pling or the like of image data by a method con- 
5 ventionally used for the driving of LCDs, and send 
the image signals B and A to the LCD 13 and LCD 
2. respectively, to thereby make the LCD 13 and 
LCD 2 display the images. As previously de- 
scribed. Fig. 6B shows the image actually dis- 
'0 played on the LCD 2. and since the LCD 2 has 
large pixels and a large display area, this image 
becomes a coarse image great in the angle of 
display field. .Fig. 5D shows the image actually 
displayed on the LCD 13, and since the p.xels of 
'5 the LCD 13 are small and the display area thereof 
is small, this image becomes a highly fine image 
small in the angle of display field 

The image signal A sent from the controller 26 
to the drive circuit 28 can also be sent as an image 
?o signal compressed by the controller 26. 

In the present embodiment, by the fact that as 
previously described, the LCD 2 and the LCD 13 
are disposed optically at the same position relative 
to the lens 4 of Fig. 2 and by the driving of the half 
?5 mirror 16. the image visually recognized by the 
observer becomes the image Fig. 5C comprising 
Fig. 6B and Fig. 5D superposed one upon the 
other. However, the center of the observer's visual 
axis is the point O of Fig. 5B, and this is substan- 
to dally coincident with the center of the image Fig. 
5D actually displayed on the LCD 13 of Fig. 2. 

The center of the visual axis detected in the 
embodiment of Fig. 2 is one detected indepen- 
dently of the size and arrangement of the pixels of 
35 the LCD 2. Fig. 7 ts a view for illustrating the size 
and arrangement of the pixels of the LCD 2 and the 
relation of the displayed position of the image 
displayed on the LCD 13 to the observer. 

In F;-; 7. m contrast with the pixels of the LCD 
-•c 2 of Fig. 2 represented tn a lattice-like form, the 
image portion displayed on the LCD 13 is indicated 
by hatching. 

In the aforedescribed embodiment, the center 
of the visual axis is determined with the size and 
-s arrangement of the pixels of the LCD 2 of Fig. 2 
not taken into account and therefore, the boundary 
when the image information Fig. 5B to be dis- 
played on the LCD 13 and the image information 
Fig. 5A displayed on the LCD 2 are superposed 
so one upon tne other does not always coincide with 
the actual boundary between the pixels of the LCD 
2. but generally overlap with some of the pixels of 
the LCD 2 Accordingly, when the samplmg pom! of 
the drive circuit 28 rests on the black signal in the 
55 image information Fig 6A. i.e.. the portion in which 
the luminance signal is 0. the black portion dis- 
played on the LCD 2 (the portion wh.ch displays no 
image) becomes larger than the portion overlap- 
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ping with the LCD 13. and an unnecessary black 
line (a non-display portion) appears m the display 
boundary portion between the LCD 2 and LCD 13 
of the image visually recognized by the observer. 
To prevent this, the vertical and horizontal lengths 
of the display portion of the LCD 13 can be set to 
integer times as great as the vertical and horizontal 
length of a pixel of the LCD 2 and the size of the 
display portion of the LCD 2 can be set to integer 
times as great as the vertical and horizontal dimen- 
sions of the LCD 13. and the visual axis position 
signal of the visual axis sensor can be converted 
into a discrete value based on the pixel arrange- 
ment of the LCD 2, in the controller 26. 

Fig. 8 shows a case where, for e^mple. the 
vertical and horizontal lengths of the display portion 
of the LCD 13 are set to two times and three times, 
respectively, as great as a pixel of the LCD 2 and 
the vertical and horizontal dimensions of the dis- 
play portion of the LCD 2 are set to six times each 
as great as the vertical and horizontal dimensions 
of the display portion of the LCD 13. When the 
output of the visua! axis sensor comes into one of 
areas partitioned at the size of the image of the 
LCD 13 on the virtual image plane 11, relative to 
the images of the pixel arrangement of the LCD 2 
{broken-line lattices) on the virtual image plane 1 1 
and area groups (solid-line lattices) partitioned at 
the size of the image of the LCD 13 on the virtual 
image plane 1 1 (for example, the point A), it is 
judged that the observer is looking at the central 
point B m that area, and as previously described, 
the original image information Fig. 5A can be de- 
composed into the image information Fig. 6A and 
the image information Fig. 5C. The digitizing of the 
above-described visual axis output is effected by 
the use of a CPU installed in the controller 26 of 
Fig. 2 and an ROM having the positional data of 
the above-mentioned area groups recorded therein. 

At this time, by properly setting the size of the 
pixels of the image sensor 24 of the visual axis 
detecting unit 35 and the imaging magnification of 
the unique point of the human eye onto the image 
sensor 24 m conformity with the size at which a 
pixel of the LCD 2 is visually recognized by the 
observer, the calculation process can be simplified 
In the present invention, the angle of rotation y of 
the optical axis of the eyeball is found from 

p*L- smtf (2a + 2b) 2-2d 

as previously described. 

p and the size of the pixels of the image 
sensor 24 can be set so that the values of 2a. 2b 
and 2d for the angle of rotation of the optical axis 
of the eyeball corresponding to the size at which a 
pixel is visually recognized by the observer may be 
integer times as great as the pixels of the image 



sensor 24 

Numerous methods of visual a*ts detection 
have beer proposed besides the method described 
m the piesent invention, but fot any of tnose meth- 
5 ods. the calculation process can be simplified by a 
similar method. Also, where as described above, 
me image by the LCD 13 for the gaze area is 
moved with the pixel pitch of the image by the LCD 
2 for wiae area display as the unit, the driving of 

io the half mirror 22 may be discrete driving cor- 
responding to a pixel of the image by the LCD 2. 
and can be effected by the use. for example, of a 
stepping motor or the like. 

The above-mentioned ratios between the verti- 

i$ cal and horizontal lengths of a pixel of the LCD 2 
and the vertical and horizontal lengths of the dis- 
play portion of the LCD 13 and the ratios between 
the vertical and horizontal lengths of the display 
portion of the LCD 2 and the vertical and horizontal 

20 lengths of the display portion of the LCD 13 are 
values set for the simplicity of description and can 
actually be set to values conforming to the purpose 
of use. Also, in the present embodiment, there is a 
case where in the boundary between the displayed 

25 image on the LCD 2 and the displayed image on 
the LCD 13. there is a discontinuous variation in 
the pixel size and it is conspicuous. In order to 
prevent this, the displayed image is made coarser 
from the center toward the marginal portion of the 

30 displayed image on the LCD 13. whereby the vari- 
ation in the pixel size in the boundary portion can 
be made inconspicuous. 

Figs. 9A and 9B illustrate the above-described 
processing. Fig. 9A shows an image to be dis- 

35 played on the LCD 13 of Fig. 2. and Fig. 9B shows 
an image subjected to the above-described pro- 
cessing and made coarser toward the marginal 
portion thereof and actually displayed on the LCD 
13. In Fig. 9B. thin solid-line lattices indicate the 

jo pixels of the LCD 13. thick solid lines indicate the 
center lines for indicating the center of the LCD 13. 
and two thin slant solid lines (A) and (B) indicate 
the boundary of the image represented by the thick 
line in Fig. 9A. As shown in Fig. 9B. the display of 

--5 the displayed image is made coarser from the 
center toward the marginal portion of the LCD 13. 
and in the boundaiy between the displayed images 
on the LCD 13 and the LCD 2 of Fig. 2, the 
coarseness of the displayed image on the LCD 13 

so is made substantially equal to the coarseness de- 
termined by the pixel size of the LCD 2. whereby 
the feeimg of physical disorder by a sudden vari- 
ation m tne pixel size can be mitigated. To carry 
out this processing, the sampling m the dnve en - 

55 cuit 29 of Fig 2 can be effected so as to be coarse 
in the marginal portion of the LCD 13 of Fig. 2 and 
fine in the central portion thereof. 
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For the smoothing of the pixels, as will 
hereinafter be Described, use may be made of a 
method whereby the image s.gnal d.splayed on the 
pixels of the gazing point portion is made into a 
signal tak.ng the weighted mean of the image sig- s 
nal roughly sampled correspondingly to the s;ze of 
tne pixels of the widely displaying LCD 2 and the 
original image signal and display is effected w.th 
the weight of the image signal wh>ch has been 
sampled more roughly toward the marginal portion w 
of the gazing point portion image being made 
great- 
Fig. 10 is a conceptual view for illustrating the 
smoothing method. 

The reference numeral 2' designates an image is 
area widely displayed by the LCD 2. and this 
image area is divided into pixels <a-i, A-j) cor- 
respondingly to the pixels of the LCD 2 <i = i 4 
J=l-5). 

The reference numeral 13' denotes the image 20 
area of the gazing point portion by the LCD 13. and 
this image area is divided into pixels (b-i. B-j) 
correspondingly to the pixels of the LCD 13 
(' = 1...7, j = i.„7). 

Image signals corresponding to the central nine 25 
P-xels {(b-3. B-3). (b-3. B-4), (b-3. B-5.). <b-2. B-3), 
<b-3. B-4). (b-3. B-5). (b-4. B-3). (b-4. B-4). (b-4. B- 
-5)} m the image area of the gaze portion are the 
original image signal as it is intactly flowed, and a 
signal having taken the mean at the weight of 1 to 30 
i of the original image signal and the roughly 
sampled image signal is flowed to sixteen pixels 
(b-2. B-2). (b-2. B-3). (b-2. B-4). (b-2. B-5). (b-2. B- 
6k (b-3. B-2). (b-3. B-6). (b-4. B-2). (b-5. B-2). (b-5. 
6-6). (b-6. B-2). (b-6. B-3). (b-6. B-4), (b-6. B-5). (b- 35 
6. B-6)} in the image area of the gaze portion 
which surround the center, and a signal having 
taken the mean at the weight of 1 to 2 of the 
or.g.nal image s.gnal and the roughly sampled im- 
age signal .s flowed to the most marginal twenty- J0 
four p.xels {(b-l. B-l), (b-1. B-2). (b-l. B-3). (b-1 
B-4). (b-l. B-5). (b-l. B-6). (b-l. B-7). (b-2. B-l) (b- 
2. B-7). (b-3. B-l). (b-3. B-7). (b-3. B-1). (b-3. B-7) 
(b-4. B-1). (b-4. B-7). (b-5. B-1). (b-5. B-7). (b-6. B- 
D. (b-6. B-7). (b-7. B-1). (b-7. B-2). (b-7. B-3). (b-7. ^ 
B-4>. (b-7. B-5). (b-7. B-6). (b-7. B-7)} .n the .mage 
area of the gazing point portion. 

The roughly sampled image signal is an .mage 
signal displayed on that pixel of the widely display- 
ing image area 2* on which rests a certain point. so 
e.g. the left upper point .n the pixels of the gazing 
pomt portion 13* (for example, (a-2. A-2) coi re- 
sponds to (b-4. B-2)). Where the left upper point m 
the pixels of the gaze portion rests on tne bouno- 
ary between the pixels of the widely displaying 55 
area 2\ the weighted mean with the image signal 
d.spiayeo on that pixel of the widely displaying 
'mage area 2* on which rests the right lower point 



m the pixels of tne gazing pomt portion 

Also, design may be made such that the piy e i 
size of the LCD 13 is actually small in the cennai 
portion and large in the marginal portion and 
gradually becomes larger from the center towatd 
the marginal portion. Fig. 1 1 shows the sizes and 
anangement of the pixels of the above-described 
LCD 13. The size of the pixels m the most marginal 
portion is set to substantially the same as the size 
of the pixels of the LCD 2 of Fig. 2. 

in the first embodiment, as previously de- 
scribed, the image information capable of being 
displayed highly finely over the whole display area 
is divided into the image information of a small 
field angle portion (the image displayed on the 
LCD 13 of Fig. 2) displayed h.ghly finely and the 
image information of a great field angle portion (the 
image displayed on the LCD 2 of Fig. 2) d.splayed 
coarsely, and in the coarsely displayed image por- 
tion, the high frequency component of the .mage 
signal has been discarded by the sampling circuit. 
In such a method, a h.gh specification is required 
of a TV camera or a computer for making the 
original image or recording means for recording 
image data, or much time and cost are required for 
making image information. 

An improvement over the first embodiment in 
the above-noted point .s a second embodiment 
shown m Fig. 12. This embodiment is almost the 
same as the first embodiment in apparatus con- 
struction, and differs from the first embodiment 
only .n the method of making the original image 
and the method of controlling the .mage. In the 
description of this embodiment, portions differing 
from those of the embodiment of Fig. 2 will de- 
scribed preponderantly and portions common to 
those of the embodiment of Fig. 2 will be briefly 
described or not described. 

Fig. 12 is a block diagram for illustrating a 
signal processing portion m the second embodi- 
ment of the present invention, and tn Fig. 12. the 
same members as those in the embodiment of Fig. 
2 are given the same reference numerals. 

The reference numeral 50 designates an image 
information generation unit comprised of a com- 
puter such as a CPU or a graphic work station, or a 
plurality of computers effecting inter-process com- 
munication or the like and effecting synchronized 
calculation processes. The .mage information gen- 
erat.on unit 50 causes an outs.de recording appara- 
tus. not shown, or a recoidmg aoparatus in the 
image information generation unit 50 to generate a 
gazing point portion .mage having data necessary 
for image information generation m the image dis- 
play area, such as. for example, the three-dimen- 
sional data of the object of display m the display * 
area, and receiving the visual axis signal from the 
detection circuit 25 and to be displayed highly 
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finely on the LCD 13 about the visual axis, and the 
background image to be displayed on the LCD 2 
other than said gazing point portion image, in a 
gazing point portion's image geneiation unit 50-2 
and a background image generation unit 50-1 in 
the image information generation unit 50. with fine- 
ness necessary for the image displayed on each 
LCD. on the basis of data necessary for said image 
information generation, sends these images to the 
drive circuits 29 and 28. respectively, and causes 
the images to be displayed on the LCD 13 and 
LCD 2. respectively. The gazing point portion's 
image generation unit 50-2 and the background 
image generation unit 50-1 generate the image 
information to be displayed with the fineness deter- 
mined by the pixel sizes of the LCD 13 and LCD 2 
and the sizes of the display portions thereof, by the 
conventionally used graphic calculation. The gazing 
point portion's image generation unit 50-2 and the 
background image generation unit 50-1 each may 
be a processor or a computer effecting said cal- 
culation, or two programs running in parallel by a 
computer. 

The controller 26 receives a visual axis signal 
from the detection circuit 25. and sends a driving 
signal to the drive circuit 27 for rotating the half 
mirror 16 on two axes. 

In the present embodiment, the division of the 
image information into the gazing point portion and 
the background portion can be accomplished by a 
calculation and operation similar to those in the 
division in the first embodiment of the present 
invention being performed within the image in- 
formation generation unit 50. Further, of the image 
to be displayed on the LCD 2 corresponding to Fig. 
6B. the image information corresponding to the 
portion (E) of. the black signal (luminance signal 0) 
superposed by the image on the LCD 13 need not 
be generated at all. but a black display signal can 
be put into that portion. Aiso, in the present em- 
bodiment, as regards said gazing point portion's 
image and said background image, besides the 
process of separating basically one image field into 
the gazing point portion and the background por- 
tion as in the signal processing m the first embodi- 
ment and the above-described present embodi- 
ment, the process of allotting the gazing point 
portion and the background portion to the lore- 
ground and the background, respectively, ts possi- 
ble. 

Figs. 13A to 13C illustrate the process of allot- 
ting the gazing point portion and me background 
portion to the foregrounc and the background, re- 
spectively. Fig. 13C shows an image which be- 
comes the background, ano this image is gen- 
erated by the background image generation unit 
50-1 of Fig. 12. Fig. 13A shows the wnole image 
information in the display area, ano a and b in this 



figure indicate persons which become the fore- 
ground in Fig 13A. c indicates an area to be 
displayed on the LCD 13 about the visual axis 
detected by the detection circuit 25. When the area 
5 c overlaps the person a oi b who is to become the 
foreground, the image of that portion of the toie- 
ground a or b falling within the area c is generated 
by the gazing point Portion image generation unit 
50-2 of Fig. 12. The other portion of the area c than 
w the foreground a or b is filled up with a black 
display signal. Fig. 13B shows the image informa- 
tion generated in the gazing point portion image 
generation unit 50-2 by the above described pro- 
cess and sent to the drive circuit 29. and the 
75 hatched portions indicate the black display signal. 
The observer can observe the background image 
Fig. 13C displayed on the LCD 2 and the gazing 
point portion image Fig. 13B displayed on the LCD 
13 while superposing these images one upon the 
20 other. When the area C does not overlap the fore- 
ground A or B, the gazing point portion image 
generation unit 50-2 of Fig. 12 generates a whole 
surface biack display signal (luminance signal 0). 
Accordingly, the observer visually recognizes only 
25 the background image. 

In the above-described process, the back- 
ground image Fig. 13C displayed on the LCD 2 
does not include the process of black-displaying 
the portion thereof superposed upon the image 
30 displayed on the LCD 13 and therefore, the ob- 
server observes the two images while completely 
superposing them one upon the other. In this case, 
it is preferable that the whole of the background 
image be displayed darkly as compared with the 
35 gazing point portion image. 

The above-described process is earned out by 
the image information generation unit 50 of Fig. 12. 
Of course, the method already described can be 
carried out for the processing of the background 
jo image in the superposed portion of the gazing 
point portion image and the background image and 
the processing to the boundary between the gazing 
point portion image and the background image. 
Fig 14 is a schematic view showing a third 
-J5 embodiment of the image display apparatus ac- 
cording to the present invention, and in Fig 14. the 
same elements as the elements shown in the first 
embodiment are given the same reference numer- 
als. This embodiment differs from the first embodi- 
so ment in that relay lenses 31 and 32 are used and 
that intermediate imaging is effected, and is the 
same as the first embodiment in the other points. 

In Fig. 14. the image information lights from the 
image display elements 2 and 13 are imaged on an 
55 intermediate imaging plane 33 by the relay lenses 
31 and 32. respectively. At this time, the intermedi- 
ate imaging of image information 13 has its imaged 
position on the intermediate imaging plane con- 
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trolled by a half minor 16 having drive means 15 
controlled by a controller and a anve circuit, not 
shown, so as to become the image information of a 
portion in which theie is the intermediate imaging 
of image information 2. and the two types of image 
information are combined on the intermediate im- 
aging plane 33. By observing the thus combined 
image through the lens 4. the observer can ob- 
serve the two images on the virtual image plane 1 1 
while superposing them one upon the other. 

Again in the present embodiment, the visual 
axis direction detecting unit 35 is disposed at a 
suitable position, whereby as in the first embodi- 
ment, the displayed position of the image informa- 
tion on the image display element 13 is controlled. 
In this manner, the observer can observe highly 
fine image information displayed near the gazing 
point on the image information from the image 
display element 2 as the background. 

Also, what combines the two images is the 
intermediate imaging plane 33. and this also leads 
to the effect that the amount of movement (the 
angle of rotation) of the movable half mirror 16 may 
be small. 

Also, in the present embodiment, the size of 
the display portion of the LCD 13. the focal length 
of the relay lens 32. the size of the pixels of the 
LCD 2. the focal iength of the relay lens 31 and the 
focal length of the lens 4 are set such that the size 
of the virtual image of the LCD 13 for highly fine 
display by the relay lens 32 and the lens 4 be- 
comes integer times as large as the size of the 
virtual image of the pixels of the LCD 2 for wide 
area display by the relay lens 31 and tne lens 4. 
and if the visual axis position signal of the visual 
axis direction detecting unit 35 is converted into a 
discrete value based on the pixel arrangement of 
the LCD 2 by the controller 26 and the displayed 
image on the LCD 13 is moved, the boundary 
between the displayed image by the LCD 13 and 
the displayed image by the LCD 2 can be made 
coincident with the boundary between the pixels of 
the image by the LCD 2. 

Further, in the present embodiment, a screen 
can be provided at the location of tne intermediate 
imaging plane 33 and the images can be combined 
on this screen and can be observed through the 
lens 4. in this case, the light of the combined 
image diffused by the screen can be observed and 
therefore, it becomes possible to make the N.A. of 
the lens 4 small, and this also leads to the effect 
that the downsizing of the appaiatus can be 
achieved. In Figs. 2 and 14. tor simplification, the 
lens 4 is shown as a single lens, but it is oesirabie 
in imaging performance that the lens 4 be a ce- 
mented lens comprising two achromatized lenses. 

Fig. 15 is a schematic view showing a fourth 
embodiment of the image display apparatus ac- 
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cording to the present invention and m Fig 15 :nc 
same elements as the elements shown m the third 
embodiment are given tne same reference numei- 
als. 

5 In this embodiment, as compared with the third 

embodiment, by the use of a combiner element 36 
such as a concave half mirror, other image in- 
formation such as a natural landscape and the 
virtual images from the image display elements 2 
w and 13 can be superposed one upon another and 
three types of image information can be observed. 
In Fig. 15. the image information lights from the 
image display elements 2 and 13 are imaged on 
the intermediate imaging plane 33 by the relay 
T5 lenses 31 and 32. respectively. The two types of 
image information combined on this intermediate 
imaging plane 33 are made into predetermined 
divergent light by the combiner element 36 com- 
prising a concave half mirror and enters the ob- 
20 server's pupil, and can be observed as a virtual 
image on the virtual image plane 1 1. 

In the present embodiment, a concave half 
mirror is schematically shown as the combiner 
element 36. but use can be made of various optical 
2S elements such as a plane half mirror, a combina- 
tion of a plane half mirror and an eyepiece, or an 
aspherical half mirror or a reflection type hologram 
lens element. 

Further, in the present embodiment, the imag- 
30 mg plane of the concave half mirror is curved and 
therefore, a so-called "gate optical system" ar- 
rangement is adopted so that the intermediate im- 
aging plane of the relay lenses 31 and 32 for the 
image information from the image display elements 
35 2 and 13 and the imaging plane of the concave half 
mirror 36 may be made coincident with each other, 
but these are suitably changed and designed in 
conformity with the characteristics of the various 
combiner elements as mentioned above. 
*o As described above, in the image display ap- 

paratus of the present embodiment, the observer 
can observe the images on the image display 
elements 2 and 13 at a time while observing an 
image such as a natural landscape through the 
js combiner element 36. and can observe the image 
information from the highly fine image display ele- 
ment 13 having its display position controlled by 
the signal from the visual axis direction aetectmg 
unit 35 always m the visual axis direction. The relay 
so lenses 31 and 32 are not the requisite constituents 
of the present embodiment. 

In the first to fourtn embodiment, for example, 
single liquid crystal display elements have been 
described as being used as the image aisplay 
55 elements 2 and 13. but it is also possible to use m 
the present invention an image display element 
provided with liquid crystal elements having Ft. G 
and B signals inputted thereto and displaying re- 
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spective color images, and combining and display- 
ing the color images by a dichioic mirror oi the ■ 
like, and a plurality of such liquid crystal elements 
and a color combining system may prefeiably be 
used particulaily as the image display element 13 5 
which is desired to display highly finely. Further, in 
lieu of the image display element 13 and the back 
light source 12. use can be made of an image 
display mechanism for driving a one-dimensional 
LED array by a video signal, rotating the half mirror 70 
16 in synchronism with -the video signal, scanning 
the LED light and displaying a two-dimensional 
image by the afterimage effect. Even if in this case, 
one-dimensional LED arrays for R, G and B. re- 
spectively, are used, the drive circuit therefor is 75 
simple and moreover, the half mirror 16 is used as 
two parts as means for scanning the light from the 
LED arrays and means for deflecting the position of 
the two-dimensional image as the afterimage and 
the other constituents are a few. and this leads to 20 
the effect that the downsizing and light weight of 
the image display apparatus according to the 
present invention can be achieved. 

Fig. 16 is a schematic view showing a fifth 
embodiment of the image display apparatus ac- 25 
cording to the present invention. This embodiment 
is an image display apparatus designed such that a 
parallax image based on the parallax of two eyes is 
observed by the respective eyes, whereby a ste- 
reoscopic image can be observed. 30 

The light from the back light source 1 is modu- 
lated by the image display element 2 such as a 
liquid crystal display element, passes through half 
mirrors 16R and 16L. is caused to enter respective 
eyes 6R and 6L by lenses 4R and 4L disposed m 30 
front of the two eyes, and forms virtual images 2'R 
and 2*L on the virtual image plane 1 1. A visual axis 
direction detecting unit, not shown, detects the 
visual axis directions 41 and 42 of the two eyes, 
and drive means 15R and 15L for the half mirrors ~-o 
16R and 16L are controlled by a controller and a 
arive circuit, not shown, and image information 
from image display elements 13R and 13L irradi- 
ated in directions 13'R and 13'L by back light 
sources 12R and 12L is imaged as virtual images. j& 
At this time, the image display element 2 displays 
image information as the background, while the 
image display elements 13R and 13L display two 
slightly different types of image information as the 
parallax image of the image 13' near the gazing so 
point e. Thus, the observer becomes able to ste- 
reoscopically observe the image of image informa- 
tion 2* which is near the gazing point e the ob- 
server wants to see. Also, in the present embodi- 
ment, head portion position detecting means, not 55 
shown, which has not been described m oetail in 
the first to fourth embodiments is mounted, for 
example, on the top portion of the apparatus. This 



position oetectmg means can utilize a suitable 
method, and is one which uses magnetism ana 
which is called a Poihemus sensor, wincn is based 
on the principle that magnetic fields o»oaucea 
time-divisionally from three coils orthogonal to one 
another which are called sources are measured by 
three similar coils (sensors) and the positions and 
postures of the sensors are found (torn the 3 x 3 
bits of information, and as other methods, mention 
may be made of an optical method of finding the 
position from the images of two cameras by the 
principle of trigonometrical survey, an ultrasonic 
method of measuring the time until a signal from 
an ultrasonic transmitter arrives at a receiver with 
respect to a combination of several transmitters or 
receivers, and calculating the position therefrom, 
and a method using a mechanical link mechanism 
disclosed by I. E. Sutherland m Proc. of the Full 
Joint Computer Conference , vol. 33. pp. 757-764 
(1968). but in the present embodiment, use is 
made of a method of mounting on the head portion 
a magnetic sensor for forming a magnetic field 
around it and detecting the magnetic field. A suit- 
able calculation process is carried out by the use 
of a head portion position signal from such head 
portion position detecting means and said visual 
axis direction detection signal and the result is 
inputted to a controller, not shown, thereby varying 
the image on the image display element 2 which is 
displaying the image information 2" as the back- 
ground. That is. when the observer is observing by 
moving his or her visual axis with his or her head 
portion remaining fixed, the image information 2* is 
not changed, but by moving or rotating his or her 
head portion, the observer changes the image in- 
formation 2' in conformity with the movement or 
rotation of the head portion. Therewith, the gazing 
point e is found by the visual axis direction detec- 
tion signal and the image information 13' near it is 
changed. 

Reference is now had to Figs. 17. 18A. 18B 
and 18C to describe the calculation process for the 
above-described head portion position signal and 
said visual axis direction detection signal and the 
relation with the then displayed images. 

Generally, when the gazing point moves to an 
object to be observed (an image one wants to see), 
the eyeballs move correspondingly thereto, and 
instead of this, it is also possible to move the head 
portion to thereby follow the object. Usually, how- 
ever, one uses these two movements to make 
one's visual axis follow. 

Heie. for simplicity, consider the matter with 
the ro;ation of center of the eyeball and the origin 
of the head portion position made coincident with 
each other. Strictly, the visual axis dnection is also 
changed by the parallel movement of the head 
portion and therefore, it is necessary correct this 
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and moreover, the origin of the head portion post- 
* tion (sensor position) and the center of rotation of 
the eyeball never coincide with each other and 
therefore, it is necessary to calibrate the relative 
position thereof in advance. 

In Fig. 17. suppose an orthogonal coordinates 
system having, x as the front of the observer. Here. 
b represents the nghtward and leftward rotation of 
the head portion. © represents the vertical rotation 
of the neck. ^ represents the vertical inclination of 
the neck, o represents the horizontal rotation of the 
visual axis, and £ represents the vertical rotation of 
the visual axis. 

Also, in Fig. 17, the reference numeral 101 
corresponds to the virtual image plane of Fig. 16 
and here, it schematically indicates a hemisphere 
always spaced apart by a predetermined distance 
R from the observer, but actually, the virtual image 
plane is a plane to which the direction 102 of the 
observer's nead portion is a normal. As a matter of 
course, this plane (virtual image plane) is some- 
times not a strict plane depending on the aberra- 
tion characteristics of the lenses 4R and 4L of the 
display apparatus, or is sometimes consciously 
curved relative to the observer in order to give the 
so-called "surrounding effect". 

At this time, as shown, the observer is gazing 
"at a point e at the left lower corner (leftward at an 
angle of oi and downward at an angle of 01) with 
his head rotated leftwardly by til from the front, his 
neck rotated upwardly by ©1 and moreover his 
neck inclined rightwardly by an angle of */1. From 
the aforedescribed assumption, the horizontal 
movement of the head portion is 0 and therefore, 
by converting a variation in the eyeball motion 
coordinates system by inclining the neck into the 
original orthogonal coordinates system, the direc- 
tion of the visual axis can be calculated and at the 
same time, the position and direction of the head 
can also be determined. 

Now. in the following of the visual axis, when 
the gazing point is in an effective field of view 
(which is generally said to be 15 degrees to right 
and left), an image can be recognized by only the 
motion of the eyeball, and when the gazing point 
moves beyond this range, the movement of the 
head also becomes necessary. 

Accordingly, in the present embodiment, the 
field angle of a highly fine image 13* neai the 
gazing point is 30 degrees and the field angle of an 
image 2* of a wide field angle is 80 degrees, but m 
an extreme case as shown m Ftg. 18A (a case 
where the gazing point e is at the left lower corner 
of the image information 2' of a wide field angle), 
the angles a and 3 of the visual axis direction each 
are 25 degrees (here, foi simplicity, the shapes of 
the images 2* and 13* are supposed as a square) 
and exceed the range of the aforementioned effec- 




tive fielo of view Accordingly, to recognize me 
suostance of the image 2' m ootail. it is necessary 
to lotate the head portion m mat direction so mat 
the gating point e may fan within the effective field 
of view 

As a result, as shown in Fig. 18C. the image 
13' must become the image of the substantially 
central portion of the image 2* and therefore, m 
conformity therewith, the display image information 
of the image 2' displaying the background image is 
changed. Fig. 18B showing the relation between 
the images 2* and 13' schematically shows the 
image display state in the course wherein the ob- 
server rotates his head portion to change from the 
display state of Fig. 18A to the display state of Fig 
18C. 

As described above, the image display appara- 
tus of the present embodiment can also detect the 
position of the head portion to thereby vary the 
image information 2' tn conformity with a change in 
the position of the head portion and highly finely 
display the portion the observer wants to see (the 
vicinity of the gazing point). Of course, this is also 
possible in the first to fourth embodiments. The 
present invention is characterized in highly finely 
displaying the portion of me image information 2' 
the observer wants to see (the vicinity of the gaz- 
ing point), and in the present embodiment, it is 
further possible to observe this portion the ob- 
server wants to see as a stereoscopic image. 

In such case, the image information to be dis- 
played on the image display elements 13R and 
13L can be made into parallax images correspond- 
ing to the respective eyes. 

Moreover, it is known that a stereoscopic im- 
age is better in image quality than a plane image of 
the same frequency band. (See. for example. NHK 
Technical Institute, R&D . No. 15. pp. 69-75. 
September 1991.) Consequently, by constructing 
the image display apparatus according to the 
present invention, there is also provided the effect 
that the quality of the image m the vicinity of the 
gazing point can be further improved. Also, as a 
matter of course, if the apparatus construction de- 
scribed m the first to fourth embodiments is pre- 
pared m two sets and respective parallax images 
are displayed to the respective eyes, there could 
be an effect similar to that of the present embodi- 
ment. In such case, the image information 2' as the 
background can also be displayed as parallax im- 
ages to the respective eyes and at this time, there 
can be realized an image display apparatus by 
which the whole image field is telescopically ob- 
served and the vicinity of the gazing point can be 
observed as a more highly line stereoscopic im- 
age. In each embodiment, for the area m which of 
the image information displaying a wide range as 
the background, the image information displaying a 
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narrow range is superposed, there may be brought 
aLOut a state m which nothing is displayed, or for 
the area tn which both overlap, the same images 
may be displayed while being superposed one 
upon the other. 5 

Also, if two liquid crystal display elements of 
the same screen size and the same pixel density 
are used and the magnifications of the relay lenses 
corresponding thereto are changed, it will be possi- 
ble to create an image of relatively rough pixels 10 
showing the background and an image of relatively 
fine pixels corresponding to the vicinity of the gaz- 
ing point. In this case, the two liquid crystal display 
elements used are the same, and this is excellent 
m massproductivity. 15 

Description will now be made of a photog- 
raphing apparatus for the images to be displayed 
by the above-described image display apparatus. 

Fig. 19A is a schematic diagram showing a first 
embodiment of the image photographing apparatus 20 
according to the present invention. 

A TV camera 1 03 having a wide angle lens 104 
mounted thereon photographs a forward field of 
view 1O6 over a relatively wide range through a half 
mirror 105. Also, a TV camera 107 uses a lens 25 
such as a standard lens or a telephoto lens to 
photograph a forward field of view 1 09 of a rela- 
tively narrow field angle through a half mirror 105. 
and records an image in an image memory, not 
shown. In the present embodiment, the former field 30 
angle is set to about 80 degrees and the latter field 
angle is set to about 30 degrees. At this time, a 
controller 1 21 for photo-taking direction has input- 
ted thereto by a suitable method a photo-taking 
direction intended by a photographer, controls a 35 
gomostage 113 having a driving unit such as a 
pulse motor and an automatically rotatable stage 
114 by a controller 117 with the aid of respective 
drive circuits 116 and 115. and photographs an 
image 106 of a wide angle. ^0 

An image at a position particularly observed by 
the photographer is controlled by a controller 120. 
and a goniostage 110 having a driving unit such as 
a pulse motor and an automatically rotatable stage 
111 ate driven by drive circuits 119 and US. -*s 
tespecttvely. to thereby introduce a highly fine im- 
age 119. 

When the photographing apparatus of the 
present embodiment and the display apparatus 
shown, for example, in Fig. 2 are used, there can 50 
be constructed a system very effective as a visual 
angle sensor m the teleexistence technique. That 
is. in the foregoing; the image photographed by the 
photographer's intention and moreover, only the 
portion observed by the photographer aie recorded 55 
highly finely, but in the display apparatus of Fig. 2. 
the position of the head and the visual axis of the 
observer (the wearer of the display apparatus) are 




detectea. This signal is inputted to tne control 
121 for photo-taking direction to theicuy pnoio- 
graph an" image of a wide fieid angic ano a highly 
fine image, and the image of a wide angle one 
output from the TV camera 103) is displayed on 
the image display element 2 of Fig 2 and the 
highly fine image (the outout from tne TV camera 
107) is displayed on the image display element 13. 
whereby the observer can observe tne image ar- 
ound the location at which the photographing ap- 
paratus of the present emoodiment is disposed as 
if he was there. 

The drive means used in the photographing 
apparatus of the present invention are not limited to 
the goniostage having a driving unit such as a 
pulse motor and the automatically rotatable stage, 
but can also be realized by the use of the half 
mirror 105 having, for example, drive means 122 
capable of relatively driven on two axes, and an 
example of such constiuction is shown m Fig. 196. 
Of course, as the construction of such drive means, 
various constructions could be employed if means 
for photographing the image of a wide angle and 
means for photographing the highly fine image can 
pan and tilt vertically and horizontally and further 
among them, the highly fine photographing means 
can pan and tilt vertically and horizontally. 

Fig. 19C is a schematic diagram showing a 
second embodiment of the image photographing 
apparatus according to the present invention. 

Again in this embodiment, it is possible to 
introduce an image reflecting the photographer's 
intention by the use of the controller 121 for 
phototakmg direction, but here, description will be 
made of a case where the image photographing 
apparatus of the present embodiment is used with 
the image display appaiatus shown in Fig 4. 

The signals from the head portion position de- 
tecting means and visual axis detecting means, not 
shown, which have been described m connection 
with Fig. 16 are inputted to the controller 121 lor 
phototakmg direction on almost real time. In this 
controller 121. the movement of the head relative 
to the detection signal for visual axis direction is 
corrected Dy the use of the head portion position 
detection signal to thereby find the direction m 
which the observer is lacing and the direction tn 
which the observer is gazing, and the whole 
photographing apparatus 1 12 is turned to the direc- 
tion in which the observer is facing by the conttol- 
ler 1 17 and drive circuits 1 15 and 1 16. whereby an 
image 106 of a wide angle is photogiaphed by the 
TV camera 103 having the wide angle lens 104 
mourned thereon. Also, m oioei to photograph im- 
ages 109R and 109L as seen from the directions of 
the right eye and left eye by TV cameras 107R and 
107L. respectively, the directions of mirrors 126R 
and 126L rotatable on two axes are controlled by 
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controllers 124R ana 124L. respectively, on the 
basis of the signal from the visual a*is detecting 
means. 

The images thus photographed are ontputted 
from respective TV camera controllers 127. 128R s 
and 128L and inputted to the aforedesenbed dis- 
play apparatus. 

That is. the image 106 of a wide angle photo- 
graphed by the present apparatus is displayed on 
the image display element 2 of Fig. 16 and forms to 
the virtual image 2*. Also, the images from the TV 
cameras 107R and 107L are displayed on the 
image display elements 13R and 13L respectively, 
of Fig. 16 and become the virtual image 13*. At this 
time, as is apparent from Fig. 19C. parallaxes are 75 
created in the respective images. Accordingly, the 
observer can observe the image in the direction 
109 photographed by the photographing apparatus 
of the present embodiment as the stereoscopic 
image 13' by the use of the display apparatus of 20 
Fig. 16. At this time, as previously described, the 
image 2' may be low in pixel density, and if the 
image pickup element density of the TV camera 
103 is the same as that of the TV camera 107. the 
image transmission from the TV camera controller 25 
127 to the image display apparatus can more en- 
hance the image compression rate than the image 
Uansmission from the TV camera controller 128 to 
thereby heighten the transfer rate. 

The image compressing method used here 30 
may be one of various methods including the con- 
ventional methods H.261. JPEG and MPEG. 

Further, to recoid the image in the present 
embodiment, the information of the direction in 
which the observer is facing when he sees by 35 
means of the display apparatus of Fig 16 and the 
direction in which he is gazing is required and 
therefore, the signal from the head portion position 
. detecting means and the visual axis detection sig- 
nal are recorded in advance on the respective -0 
images, and the images coinciding with the head 
portion position and the visual axis direction when 
the observer has observed wearing the display 
apparatus are random-accessed from an image 
memory 129. whereafter they can be inputted to -5 
the respective display elements of the display ap- 
paratus. 

This invention relates to an image display ap- 
paratus navmg first image generating means for 
cisDiaymg first image information by a plurality of to 
pixels emitting light, second image generating 
means fc: displaying second image information by 
a plurality of pixels emitting hght. the second image 
information being part of the first image informa- 
tion, visual axis detecting means for detecting an 55 
observer's visual axis direction, optical means for 
combining a first image from the first image gen- 
eratmg means and a second image from the sec- 



ond image generating means, moving the seronc 
'image relative to the first image m conformity with 
an output Signal from the visual axis detecting 
means and turning it to the obseiver*s pupii the 
marginal portion of the second image being maae 
substantially coincident with the boundary between 
the pixels of the first image when the images are 
combined, and image display changing means lor 
changing the first image information and the sec- 
ond image information on the basis of the output 
signal of the visual axis detecting means. 

Claims 

1. An image display apparatus comprising: 

first image generating means for display- 
ing first image information by a plurality of 
pixels emitting light; 

second image generating means for dis- 
playing second image information by a plural- 
ity of pixels emitting light, said second image 
information being part of said first image in- 
formation; 

visual axis detecting means for detecting 
an observer's visual axis direction; 

optical means for combining a first image 
from said first image generating means and a 
second image from said second image gen- 
erating means, moving said second image rel- 
ative to said first image in conformity with an 
output signal from said visual axis detecting 
means and turning it to the observer's pupil, 
the marginal portion of said second image 
being made substantially coincident with the 
boundary between the pixels of said first im- 
age when said images are combined: and 

image display changing means for chang- 
ing said first image information and said sec- 
ond image information on the basis of the 
output signal of said visual axis detecting 
means. 

2. An apparatus according to Claim 1. wherein 
the size of said second image is integer times 
as great as the interval between the pixels of 
said first image when said images are com- 
bined. 

3. An apparatus according to Claim 1. wherein 
said first image information in an area wherein 
said first image and said second image are 
superposed one uoon the other when said 
•mages are combined is said second image 
information. 

4. An apparatus according to Claim 3. wherein 
the area of said images superposed one upon 
the other is not displayed by said first image 
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generating means 

5. An apparatus according to Claim 1, wherem 
the pixel density m the central portion of said 
second image is higher than the pixel density 5 
of said first image and the pixel density in the 
marginal portion of said second image is sub- 
stantially the same as the pixel density of said 

first image when said images are combined. 

jo 

6. An apparatus according to Claim t. wherein 
the luminance in the central portion of said 
second image is higher than the luminance of 
said first image and the luminance in the mar- 
ginal portion of said second image is substan- is 
tially the same as the luminance of said first 
image when said images are combined. 

7. An apparatus according to Claim 1. wherem 

said first and second images are combined on 20 
an intermediate imaging plane through respec- 
tive relay lenses, are enlarged as virtual im- 
ages and are turned to the observer's pupil. 

8. An apparatus according to Claim 7. wherein 25 
said relay lenses have different imaging mag- 
nifications. 



eraimg means, moving said second imago rot- 
ative to said first image m conformity with an 
output signal from said visual axis detecting 
means and turning it to the observer's pupil, 
and 

image information changing means for 
. changing said first image information and said 
second image information on the basis of the 
output signal of said visual axis detecting 
means and the output signal of said head 
portion position detecting means. 

13. An apparatus according to Claim 12. further 
comprising an image memory having recorded 
therein the image all around a predetermined 
position, and wheiem a part of said image 
memory is inputted as said first image in- 
formation to said first image generating means 
m conformity with the output signal of said 
head portion position detecting means. 

14. An apparatus according to Claim 12. wherem 
said first image information in an area wherein 
said first image and said second image are 
superposed one upon the other when said 
images are combined is said second image 
information. 



9. An apparatus according to Claim 1. wherem 

said image display apparatus is provided cor- 30 
respondmgly to each of the observer's left and 
right eyes. 

10. An apparatus according to Claim 9. wherein 

said second image generating means corre- 35 
sponding to the left and right eyes display 
parallax image information corresponding to 
the respective eyes. 

11. An apparatus according to Claim 9. wherein jo 
said first image generating means is used m 
common. 

12. An image display apparatus comprising: 

first image generating means for emitting js 
light and displaying first image information; 

second image generating means for emit- 
ting light and displaying second image rnfoi- 
mation. said second image information being 
part of said first image information; r j0 

visual axis detecting means for detecting 
an observer's visual axis direction. 

head portion position detecting means foi 
detecting the position of the ODserver's head 
portion; 55 

optical means for combining a first image 
from said first image generating means and a 
second image from said second image gen- 



15. An apparatus according to Claim 14. wherem 
the area of said images superposed one upon 
the other is not displayed by said first image 
generating means. 

16. An apparatus according 10 Claim 12. wherem 
said first and second images are combined on 
an intermediate imaging plane through respec- 
tive relay lenses, are enlarged as virtual im- 
ages and are turned to the observer's pupil. 

17. An apparatus according to Claim 16. wherem 
said relay lenses have different imaging mag- 
nifications. 

18. An apparatus according to Claim 12. wherem 
said image display apparatus is provided cor- 
respondmgly to each of the observer's left and 
right eyes. 

19. An apparatus according to Claim 18. wherem 
said second image generating means corre- 
sponding to the left and right eyes display 
parallax image intoimation corresponding to 
the respective eyes. 

20. An apparatus according to Claim 19. wherein 
said first image generating means is used m 
common. 
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21. An image photographing apparatus for an im- 
age display appaiaius fo» displaying, on the 
basis of output signals from visual axis delect- 
ing means for detecting the visual axis dnec- 
tion of an observei who observes a first image 5 
from first image generating means for display- 
ing first image information and head portion 
position detecting means for detecting the po- 
sttion of the observer's head portion, a com- 
bined image of said first image and a second to 
image from second image generating means 
for displaying second image information which 
is part of said first image information, compris- 
ing: 

first image pickup means for obtaining said is 
first image information; 

second image pickup means for obtaining 
said second image information; 

means for controlling the photo-taking di- 
rection of said first image pickup means in 20 
conformity with the output signal from said 
head portion position detecting means; and 

means for controlling the photo-takmg di- 
rection of said second image pickup means in 
conformity with the output signal from said 25 
visual axis detecting means. 

-22. An apparatus according to Claim 21. further 
comprising means foi transmitting the image 
information obtained from said first image pic* 30 
kup means to said image display apparatus 
while making the compression rate of said 
image information highei than that of the in- 
formation obtained from said second image 
pickup means. J5 

23. An apparatus according to Claim 21. wherein 
said image photographing apparatus is pro- 
vided correspondingly to each of the observ- 
er's left and right eyes. ^0 

24. An apparatus according to Claim 23. wherem 
said first image pickup means is used m com- 
mon. 

25. An image display method comprising: 

the first image generating step of display- 
ing first image information by a plurality of 
pixels emitting light; 

the second image generating step of dis- so 
playing second image information by a plural- 
ity of pixels emitting light, said second image 
information being part of said first image in- 
formation; 

the visual axis detecting step of detecting 55 
an ooserver's visual axis direction: 

the step of combining a firs: image from 
said first image generating step and a second 
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image from said second image generating 
step, moving said second image relative to 
said first image m conformity with an output 
signal from said visual axis detecting step and 
turning it to the observer's pupil, the marginal 
portion of said second image being macie sub- 
stantially comcioent with the boundary between 
the pixels of said first image when said images 
are combined: and 

the image display changing step of chang- 
ing said first images information and said sec- 
ond image information on the basis ol the 
output signal of said visual axis detecting step. 

26. A method according to Claim 25. wherein the 
size of said second image is integer times as 
great as the interval between the pixels of said 
first image when said images are combined. 

27. A method according to Claim 25. wherein said 
first image information in an area wherein said 
first image and said second image are super- 
posed one upon the other when said images 
are combined is said second image informa- 
tion. 

28. A method according to Claim 27. wherein the 
area of said images superposed one upon the 
other is not displayed in said first image gen- 
erating step: 

29. A method according to Claim 25. wherein the 
pixel density in the central portion of said 
second image is higher than the pixel density 
of said first image and the pixel density in the 
marginal portion of said second image is sub- 
stantially the same as the pixel density of said 
first image when said images are combined. 

30. A method according to Claim 25. wherein the 
luminance in the central portion of said second 
image is higher than the luminance of said first 
image and the luminance m the marginal por- 
tion of said second image is substantially the 
same as the luminance of said first image 
when said images are combined. 

31. A method according to Claim 25. wherein said 
first and second images are combined on an 
intermediate imaging plane through respective 
relay lenses, are enlarged as virtual images 
and are turned to the observer's pupil. 

32. A method according to Claim 31. wherein said 
relay lenses have different imaging magnifica- 
tions. 
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34 



33. A method according to Claim 25. wherein said 
steps are ptoviaed conespondmgiy to each of 
the observer's left and right eyes 

34. A method according to Claim 33. wherein said 5 
second image generating steos corresponding 

to the left and right eyes display parallax im- 
age information corresponding to the respec- 
tive eyes 

70 

35. A method according to Claim 33. wherein said 
first image generating step is used in common. 

36. An image display method comprising: 

the first image generating step of emitting 75 
light and displaying first image information: 

the second image generating step of emit- 
ting light and displaying second image infor- 
mation, said second image information being 
part of said first image information; 20 

the visual axis detecting step of detecting 
an observer's visual axis direction; 

the head portion position detecting step of 
detecting the position of the observer's head 
portion; 25 

the step of combining a first image from 
said first image generating step and a second 
image from said second image generating 
step,, moving said second image relative to 
said first image in conformity with an output 30 
signal from said visual axis detecting step and 
turning it to the observer's pupil: and 

the image information changing step of 
changing said first image information and said 
second image information on the basis of the 35 
output signal of said visual axis detecting step 
and the output signal of said head portion 
position detecting step. 

37. A method according to Claim 36. further com- jo 
prising the image memory step of recording 
image information ail around a predetermined 
position, and wherein part of the image in- 
formation recorded by said image memory 

step is inputted as said first image information 
to first image generating means in conformity 
.with the output signal of said head portion 
position detecting step. 

38. A method according to Claim 36. wherein said 50 
first image information m an area wherein said 

first image and said second image are super- 
posed one upon the other wnen said images 
are combined is said second image informa- 
tion. 55 

39. A method according to Claim 38. wherein the 
area of said images superposed one upon the 



otne' is not displayed by said fust image gen- 
eta::ng step 

40. A method according to Claim 36. wneretn said 
first and second images are comomed on an 
intermediate imaging plane through respective 
relay lenses, are enlaiged as virtual images 
and are turned to the observer's pupil 

41. A method according to Claim 40. wherein said 
relay lenses have different imaging magnifica- 
tions. 

42. A method according to Claim 36. wherein each 
of said steps is provided correspondingly to 
each of the observer's left and right eyes. 

43. A method according to Claim 42. wherein said 
second image generating steps corresponding 
to the left and right eyes display parallax im- 
age information corresponding to the respec- 
tive eyes. 

44. A method according to Claim 42. wherein said 
first image generating step is used in common. 

45. An image photographing method for an image 
display method of displaying, on the basis of 
output signals from the visual axis detecting 
step of detecting the visual axis direction of an 
observer who observes a first image from the 
first image generating step of displaying first 
image information and the head portion posi- 
tion detecting siep of detecting the position of 
the observer's head portion, a combined image 
of said first image and a second image from 
second image generating means for displaying 
second image information which is part of said 
first image information, comprising: 

a first image pickup step tor obtaining said 
first image information; 

a second image pickup step for obtaining 
said second image information: 

the step of controlling the photo-taking di- 
rection of said first image pickup step in con- 
formity with the- output signal from said head 
pomon position detecting step; and 

the step of controlling the photo-taking di- 
rection of said second image pickup step m 
conformity with the output signal from said 
visual axis detecting step. 

46. A method according to Claim 45. further com- 
prising the step of transmitting the image in- 
formation obtained from said fust image pickup 
step to an image display apparatus while mak- 
ing the compression rate of said image in- 
formation higher than that of the image in- 
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formation obtained from said second image 
pickup step. 

47. A method according to Claim 45. wherein each 

of said steps is provided correspondingly to 5 
each of the observer's left and right eyes. 

48. A method according to Claim 46. wherein said 
first image pickup step is used in common. 

w 
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FIG. 3 A 
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FIG. U 
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FIG. 5 A 
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FIG. 6B 
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FIG. 8 
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FIG. 11 
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